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@ Automatic Image creation by merging text image and form tnriage. 

@ Input images of a fonm (250) and of a text 
(260) are used to automatically obtain an output 
image (300) showing the form with the text in a 
field (302). A processor receives data defining 
the input images (250,260), automatically oth 
tains positioning data indicating positions of 
boundaries around the field (254) so that the 
text (262) can be positioned within the field, and 
automatically obtains data defining the output 
Image. The positioning data can be automati- 
cally obtained by operating on plain images of 
the fonn and of the text ; by operating on 
images of the form and of the text that have 
been marked to indicate the field and the text; 
or by operating on images of the form and of the 
text that include machine readable information 
indicating the positions of the field and the text. 
Skew can be corrected before merging a text 
with a fonm, and text can be scaled (294,296) 
and reformatted if necessary to fit into a field. If 
the input images include more than one field on 
the fomn and more than one text segment, the 
processor can obtain linking data indicating a 
respective text segment for a field. The linking 
data can also indicate a field that is a continua- 
tion of another field or a text segment that is a 
continuation of another text segment. In case of 
overrun or insufficient text to fill a field. The 
processor could be in a fax server, a digital 
copier, or a computer with a scanner and a 
printer. 
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sJ^a^r iran^^^^^^^ i-ges that inCude text 

such as vouchers or certificates^^' att or^^^^^^^^^^^ re^ToVrst r'""?'-'^"''^'^"^ '"''^''"^^ ^"^^ 
As shown and described in relation to Figs 1 and 2 of f^""" ""P^" 

5 at a predetermined position on a platen and is scanned ^he slnl " "^'''"^"^ f>^^^^ 

Then, a page containing a voucher for whici a SrSflcatl r. rln ^T'^' '"^^'^^'^ in memory, 

determined position on the platen, a potion numJl^^^ tl^l^^^^^^^^^^ ' '"^"^ ^"'^ "'^-^ « 

voucher is scanned, and the signals are storeZmeml JtnTe '^^^^ P^"«'- 
control section reads the data for the certif iite foTm from 11^ ^^^''^'^^'^ fo^'"- A 

10 from the data from the book, and transcnb«?he Inrl ^.^ '^^'^^ ^" '^^"''^<i Portion 

certificate form to prepare data fofa "ra^LaT^o^^^^^ ^-'a'orthe 
icate is then transferred to a printer aXinted rh Z^^^^^^ ""P^^«'°" 

pear on a l,st and are placed a^a pred^l mS IS ^"''^'^riptions) are stacked in an order to ap- 

Platen scans .he indM^ual slips anTifertrSrst;!!^^^^^^ ^'-^ 
reads the data for the list for,? from the f^B^Z^rou^T!^°" ^ section 

individual Slip data, and transcribes them ThToveVlav^ri^L'^^T"^'^ designated portions of the , 

Suganuma et al. indicate that input order of ^vtuaJslbs ma^ h ""^^ ^^^'^^ • 

marks or numerical figures placed on the inlidS ' ''«'«^'"'n«<l the system by recognizing 

.echni^lTofe;::"^^^^^^^^^^^ 

Shown and described in relation to Fig. 1 o^Greene 1^8 ?d J/ Tf'^ "^""^ ""^""'"^ ^""^ Processing. As 
for a numerical value, and for a wntten vll Ar!hl !! ''^^^^^ 

fields can be scanned to obt Inr^JstanaM^^^^ ^ °^<5--« -"'b the 

account number and merged with o.herfSgene-tfa sJaS^^^^^^^^ ''^^ ''V 

copy statement that includes payee and amount f^l^Thlt^T^^^ a hard 

documents. Greene "211 descriLs a docZnt rr^^s Jlr^^^^^^^^ 

relation to Fig. 1. document processmg apparatus that includes a statement generator in 

thatca^^fe'srecli'^ir^^^^^^^ 

displays a job ticket with a mlrge sel'cl^ \hat can be acJateS to d^^^^^^^^ *° "'^Z P''"*'"^ 
.con. Actuating the merge icon displays a merqe Sarv <irrt n h ^ ^ ^P**"^' «'=°''««=«rd with a merge 
to open the merge library card f ile shLrn'Zgi i a ZZ vt '""^^.f ""^^^^ '^^^ be actuatL 

stores merge images that may be selected forrelT Jh r ' -^ 'l^- "^^'^^ "'^"'^ ""^-^^y 
headings, form paragraphs, etc. Figs l^irreTate to Tm ^ f- " P"=*"^«^- 

merge item on a print is Shown and Lcn^Jdintatio^^^^^^ P^''"^"'^'' P-'*-"-^ 

to F,gs. 18 and 19. a document may be scanned t^conver the ^"^^ 'lescribed in relation 

and paste process can be performed to ob^n a dfeZTa 1 'nl . * 'Tf '° """^^ ^'^"^'^ ^"^ « 
its size or orientation. Then the document is^sLtned^ob^^^^^^^^^^ ''""^^ 

The invention deals with problems that Irise inTntl?. / ! '"^'^e item, 

one or more fields for insertio^ of S " ''^'^ °^ ^ ^any forms include 

the --sTvLS^^^^^ is only possible, however, when 

arelk^y^r^^^^^^^^^ 

spaces, or other format features so that the text aooLTJL TJ \ ''^^ '"^^'^"9 "margins, blank 

The formatted text can then be printed onto ^=^^0' the foJ^Th^tlcH ' "'"T '° 
because several iterations may be required to cor^rcJ^^sSrth^to^ Sn^h^^^^^ '"^"'"^'^"^ 
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Text can be inserted in a field of a form available as a paper document by manually typing or writing the 
text. If the text is available in an existing document, however, retyping or rewriting is inefficient and may intro- 
duce errors. 

Text that is available in a paper document or that can be printed onto paper from ah electronic document 
can also be cut and pasted into a field of a form available as a paper document. The cut and pasted version 
of the form can then be copied. This technique can be inefficient and messy if small segments of text, such 
as words, must be cut and pasted. In addition, this technique may introduce artifacts such as skew of a pasted 
segment or lines or shadows indicating edges of pasted segments. 

It is an object of the invention to provide a technique for inserting text in a field of a form that avoids these 
problems. 

According to the present invention, there is provided a method of operating a system that includes image 
input circuitry for receiving data defining images as input; image output circuitry for providing data defining im- 
ages as output; and a processor connected for receiving data defining images from the image input circuitry 
and for providing data defining images to the image output circuitry; the method comprising operating the proc- 
essor to receive Input image data from the image input circuitry; the input Image data defining a first image 
set that shows a form that includes a first field and a second image set that shows a text; operating the proc- 
essor to use the input Image data to automatically obtain form image data, text image data, and positioning 
data; the form image data defining an image of the form from the first image set; the text image data defining 
an image of the text from the second image set; the positioning data Indicating positions of boundaries around 
the first field so that the text can be positioned within the first field; and operating the processor to use the 
form image data, the text Image data, and the positioning data to automatically obtain output image data and 
to provide the output image data to the image output circuitry; the output image data defining a third image 
set that shows the text and the form, the third image set showing the text positioned within the boundaries 
of the first field. 

The technique of the invention has the effect of merging an image of the form with an image of the text, 
and has the advantage of permitting automatic form filling efficiently and easily. 

The technique can be implemented by receiving input image data and using the input image data to auto- 
matically obtain output image data. The input image data define a first image that includes a form. The input 
image data also define a second image that includes text The output Image data define a third image that 
includes the form from the first image with the text from the second image within a field. 

To obtain the output Image data, the technique automatically obtains form image data, text image data, 
and positioning data from the input image data. The form image data define an image of the form. The text 
image data define an image of the text The positioning data indicate positions of boundaries of a field within 
the form so that the text can be positioned within the field. The technique then uses the form image data, the 
text image data, and the positioning data to automatically obtain the output image data such that the third image 
Includes the form with the text positioned within the field's boundaries. 

The positioning data could be automatically obtained in several ways. For example, the input image data 
could be used to obtain data Indicating pixels in an area that is enclosed by lines, that meets a blank area cri- 
terion, and that is sufficiently large to be filled; the positions of boundaries could be obtained by finding a bound- 
ing box for the pixels. The form could include a mark inside a field to be filled, and the mark could be detected, 
making it unnecessary to apply a blank area criterion; the technique could remove the mark in obtaining the 
form image data. Alternatively, the form could include machine readable information indicating a field's boun- 
daries. 

The technique can be implemented with a fax server or other machine that can transmit and receive data 
defining images. For example, the input image data can be transmitted to the fax server from a fax machine. 
The input image data can define a sheet indicating a merge operation. The Input image data can also include 
information indicating a separation between an image set defining a form and an image set defining a text. 
The fax server can use the input image data to obtain the form image data, the text image data, and the pos- 
itioning data. Then the fax server can use the form image data, the text image data, and the positioning data 
to obtain the output image data. The fax server can transmit the output image data to the fax machine where 
it is printed. 

Similarly, the technique could be Implemented with a copying machine that can obtain and provide data 
defining images, such as through scanning circuitry and printing circuitry. Again, the copying machine's proc- 
essing circuitry can use input image data from the scanning circuitry to obtain the the form image data, the 
text image data, and the positioning data. Then the processing circuitry can use the form image data, the text 
image data, and the positioning data to obtain the output image data, which it provides to the printing circuitry. 

Finally, the technique could be implemented with a computer, such as a workstation or personal computer, 
that can transmit and receive data defining images through input/output devices, such as a printer and a scan- 
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ner. As above, the computer-s central processing unit (CPU) can use input image data from the scanner or 
other .mage input device to obtain the form image data, the text image data, and the positioning data. Then 
the CPU can use the form image data, the text image data, and the positioning data to obtain the output image 
data, which it provides to the printer or other image output device. 

A method and system in accordance with the invention will now be described, by way of example with 
reference to the accompanying drawings, in which:- k . mi 

Fig. 1 is a schematic flow diagram showing how two images, one with a form and the other with text can 
be merged to position the text in a field in the form. 

Fig. 2 is a flow chart showing general acts in merging images of a form and text as In Fig 1 
Fig. 3 IS a schematic block diagram showing a software product that can be accessed by a system to au- 
tomatically merge images as in Fig. 2. j y = lu au 

and Si ^ schematic block diagram showing a fax server that can automatically merge images of a form 

« ^.S^' ^ ^ ^"^^'^^^'^ «<«9ram showing a digital copier with processing circuitry that can automatically 
IS merge images of a form and text. ' 

of a fd?m^ind tSt^'"^*'' ^ computer with a CPU that can automatically merge images 

plain'^Lg^onexl.^'"^'"' """^ ^"""^'"^ ^'^ automatically merging a plain image of a form with a 

fi.JS'JJXL^l'"^^'' ^"to'^atically merging an image of a form in which a 

field IS marked with an image of text in which a text segment is marked 

fnr J^5' ^ ^"^^""^^^ '"agraii showing acts in automatically merging an image of a form with encoded 

form Identifying information with an image of text with encoded information indicating a text segment 

^ Fig. 1 0 IS a schematic flow diagram showing general acts in automatically merging images as in Figs. 7- 

f ^\ V ^"^^"^^''^ ^'^"^ °^ ^" '"^^9^ °^ 'e'^ vvhich spacings between paragraphs provide information 

for linking text segments to fields; » f y iiiiumwuon 

=. H K-^^ ^ schematic view of a sequence of images including a form for requesting a fill form operation 
and in which marks in fields and in text indicate continuations between fields and text segments for linking 

Fig. 1 3 IS a schematic view of a sequence of images including a cover sheet requesting a fill form operation 
and a separator sheet, and in which encoded information indicate types of fields and text segments for linking. 

A. General Features 
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Figs. 1-3 .lliistrate general features of the invention. Fig. 1 is a schematic flow diagram showing how two 
^nages. one with a form and the other with text, can be merged to automatically obtain an image of the form 
with the text posrtioned in a field. Fig. 2 is a flow chart showing general acts in automatically merging two i^ 
ages as in Fig. 1. Fig. 3 is a schematic block diagram showing a software product that can be accessed to 
automatically merge two images as in Fig. 2. 

Image 1 0 in Fig. 1 shows a form with field 12. Field 12 is Illustratively a blank area surrounded by a line 

wi^hr„ TtT ^ ! .!.\''°'"^""^"'^' '''^"""^^ '^""''^^y around fields to Indicate boundaries 

wrthin which text should be inserted. Many forms also include a small amount of text within or just outside a 
field s boundaries to indicate the type of text to be inserted in the field. 
Image 20 in Fig. 1 shows a text, "Words to go into field." 

Data defining image 30 in Fig. 1 are obtained automatically from data defining images 10 and 20 Data 

are ohtll/n T"" Trff- '° ^^'^ ^^^""'"9 '"^^^^ ^^"^ '^^^'"'"a t^e text in image 20 

are obtained from data defining image 20. Positioning data indicating boundaries of field 12 are also obtained 
from data defining image 10 so that the text can be positioned within field 32 in image 30 as shown 

r,^J^!^^ltV° i ^'f ^ T^^^^ '""^^^ ^^'^ ''^""'"^ ^ set that Shows a form with a 

field, Ike image 10 in Fig. 1. and a second image set that shows text, like image 20 in Fig. 1. The act in box 

11 .IT H ^ ""^^^ "^^^ ^" ""396 data defining an Image of 

the text and positioning data Indicating positions of the field's boundaries. The act in box 44 uses the data 

wStJe'li^'^ t '° T""^^''^ '^^^'■"'■"g a third image set that shows the form with the text 

within the field s boundanes. like image 30 in Fig. 1. 

Fig. 3 shows software product 60, an article of manufacture that can be used In a system that Includes 
components like those shown in Fig. 3. Software product 60 includes data storage medium 62 that can be ac- 
Snfr« ""^ ^''^^ 64. Data storage medium 62 could, for example, be a magnetic me- 

dium such as a set of one or more floppy disks, an optical medium such as a set of one or more CD-ROMs 
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or any other appropriate medium for storing data. 

Data storage medium 62 stores data that storage medium access device 64 can provide to processor 66. 
Processor 66 is connected for receiving data defining images from image input circuitry 70. The data could be 
obtained from facsimile (fax) machine 72; from scanner 74, which could be the scanner of a digital copier or 
5 an input/output device of a computer; from editor 76. which could be a forms editor or other interactive image 
editor controlled by user input devices such as a keyboard and mouse or a pen- or stylus-based input device; 
or from network 78. which could be a local area network or other network capable of transmitting data defining 
an image. 

Processor 66 is also connected for providing data defining images to image output circuitry 80. The data 
10 could in turn be provided to fax machine 82. to printer 84. to display 86. or to network 88. Printer 84 could be 
the printer of a digital copier or an input/output device of a computer. 

In addition to data storage medium 62, software product 60 includes data stored by storage medium 62. 
The stored data include data indicating image receiving instructions 90, which can be executed to perform the 
act in box 40 in Fig. 2. The stored data also include data indicating response instructions 92. Response in- 
15 structions 92 can be executed to automatically obtain form image data, text image data, and positioning data, 
as in box 42 in Fig. 2. Response instructions 92 can also be executed to automatically obtain data defining a 
merged image, as in box 44 in Fig. 2. 

B. Implementations 

20 

The general features described above could be implemented in numerous ways on various machines to 
obtain merged images. 

1. Machines - 

25 

Figs. 4-6 illustrate three machines in which the invention can be implemented by modifying conventional 
components. Fig. 4 shows a fax server implementation. Fig. 5 shows a digital copier implementation. Fig. 6 
shows a computer implementation. 

In Fig. 4, fax machine 120 is used to transmit data defining an image set to fax server 122. The transmitted 
30 data define image 1 30 showing a first form, image 1 32 showing a second form, and image 1 34 showing a text. 
The first form includes a first field with check box 136 marked to request a fill form operation merging the 
second form and the text. 

Fax server 122 uses data defining image 130 to obtain fill form request data indicating a request for a fill 
form operation, based on information in the data defining image 130 indicating that check box 136 is marked. 

35 Then, responding to the request, fax server 122 uses data defining image 132 to automatically obtain form 
image data defining an image showing the form. Fax server 122 also uses data defining image 134 to auto- 
matically obtain text image data defining an image showing the text. Fax server 122 also uses data defining 
images 130, 132, and 134 to automatically obtain positioning data indicating positions of txsundaries around a 
field on the form so that the text can be positioned within the field. Fax server 122 uses the form image data, 

40 the text image data, and the positioning data to automatically obtain data defining an image showing the form 
with the text within the field. Fax server 1 22 transmits this data to fax machine 120. where it is printed to obtain 
image 140. 

In Fig. 5, digital copier 160 includes scanning circuitry 162 that obtains data defining image 164 showing 
a form and image 166 showing a text. User interface circuitry 170 includes touch sensing device 172. which 

45 can be a push button, a heat or pressure sensitive element, a capacitance sensing element, or other device 
for sensing a touching action. When a user touches device 172. user interface circuitry 170 provides touch 
data indicating that device 172 has been touched. 

Processing circuitry 160 uses the touch data to obtain fill form request data indicating a request for a fiN 
form operation. Then, responding to the request, processing circuitry 180 uses data defining images 164 and 

50 166 to automatically obtain form image data defining an image showing the form, text image data defining an 
image showing the text, and positioning data indicating positions of boundaries around a field on the form so 
that the text can be positioned within the field. Processing circuitry 180 uses the form image data, the text 
image data, and the positioning data to automatically obtain data defining an image showing the form with the 
text in the field. This data is provided to printing circuitry 190 for printing of image 192. 

55 In Fig. 6. scanner 200 obtains data defining image 202 showing a form and 204 showing a text This data 

is provided to CPU 210 of a computer, such as a workstation or personal computer. CPU 210 is connected for 
receiving data indicating user actions from keyboard 21 2 and mouse 214 and for providing data defining images 
to display 216. CPU provides an image that includes selectable unit 220 that can be selected to request a fill 

5 
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CPU 210 uses the signal to obtain fill form request data indicating a request for a fill form operation In 
response to the request. CPU 210 uses the data defining image 202 and image 204 to IZaS ^06^ n 
form image data def.nmg an image showing the form, text image data defining an image shoS he tSi 
and posrt.on.ng data indicating positions of boundaries around a field on the form so that the te« can be doJ 
foned within the field. CPU 210 then uses the form image data, the text »T,ageli and the 



2. Merging 



ica.lv oh^l;12 f ""'^'^^^^y^ ^"^-^ ""age data, text image data, and positioning data can be automat- 
.cally obtained and used to automatically obtain data defining an image showing a form with text in a f ie d Fia 
15 7 shows an implementation in which form image data, text image date, and portioning data ^robtaineirfrom 

TmLp Hl J? ^^"^ been marked by a user. Fig. 9 shows an implementation in which form 

image data, text image data, and positioning data are obtained from images that indude machine readable en 

.0 orxr^dTrr^g" ^ ^^^^ '-^^^^ ^-^-^-^ - - 

In Fig. 7, image 250 shows a form that includes fields 252 and 254 Field 252 is a reiativpiu cmaii f i^.w 

fmagei 250 and'eo " ''"""'"^ '^'"^""^^ ''^ '° <^«^'"-9 

25 Data defining image 250 can be used to obtain data defining image 270 a version of imaoe 2>5n in «,hirh 

ra:K%:rc::b:'nfor^^^^^^ 

mask produced by morphological techniques resembling those described in copending EP-A-0 543 599 and 
referred to as "the image morphology application." For example, data defining iLge 27rcould S p^ducS 
by performing first, a morphological edge fill operation that fills with black pixelsLard f^m the edls^f 
the 'mage unhl a closed boundary is reached; the edge filled image could then be inverted obtot black 
pixels within fields 252 and 254 and white pbcels elsewhere, as in image 270 

Small features within otherwise solid areas could be eliminated by morphological open operations on data 
n^T T'' f white pixels. To find a Z.le? 

at Inllrn ' fTT'' " «" be performed within the larger f iefd bSTng 

at any location; ,f the fill operation fills less than the larger field, the area filled can be removed from thi Zer 
field and an o her fill operation can be performed on the non-filled area to find other smaller fields If it fs de 

tS rdert^h^r^ni^ofr^ '-^^^^^^ - — = 

mo Jhlo?!i ^"1""^ '^""""T^ ""^^ """'^ ^ ^'^"^'^ boundaries are not continuous. Prior to edge filling a 
morphological dose operation could be performed to dose gaps in a field's boundaries Morphol«,i«i "ech 
niques could alternatively be used to find a bounding box for each connected componentin mage 2S Se"r 

llntsTh! T'^' *° '^"""'"^ large /connected co^ 

cLTd tt n h H f T '^^f ''^'^ boundaries are not continuous. The boundSg boxTs 

could then be used to obtain data defining image 270. "uunumg ooxes 

Data defining image 270 can then be used to obtain data defining image 280. a version of imaoe 270 in 

of ?hrh f fr^-" "t''^^ P«els by applying a blank area criterion and a size threshold to^aS 

suIJ «stf ^TT Tk'T'^ '^''"'"5 280 can then be used to obtain posiSoning data 

such as data indicating the horizontal and vertical bounds of segment 282 posmoning aata. 

inH ^/TV °^ "^[I^ """"^ "^^"^ *° discriminate segment 282 from any segments that alreadv 

Se oiS toS^?-. "r"''"' '^''^^'^ ^ -"'^ coumedrdTpropol"^ 

thrLshoJT o H .P'^'^'V" "^'"^"^ °'^'«'"«<1- This proportion could then be compared with a 

55 dSe to ftr" " " predominantly white. If not. the segment would not be a ^an 

of co^n?j!r™ « H "T'^ f thresholds could be used. For example, the size threshold could be a number 
of contiguous b ad< pixels or a dimension, such as 25mm. in each direction. A segment below threshold Tuch 
as segment 274. would not be a candidate to be filled with text because of its smlll st The Sold co"d 
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be applied to a count of black pixels in each segment or to measurements of height and width of the largest 
rectangle that fits within each segment. Instead of using a default size threshold, data indicating the size 
threshold could be obtained from the userthrough marks on a form, through user i nterface circuitry, or through 
a graphical user interface. 

5 Similarly, data defining image 260 can be used to obtain data defining image 290. a version of image 260 

in which pixels within a bounding box around text 262 are all black and all other pixels are white. Image 290 
could be a mask produced by techniques resembling those described in the image morphology application. 
Pixels in segment 292 in image 290 could be discriminated from other pixels by applying a text area criterion. 
A bounding box could then be obtained around segment 292. and the bounding box could be filled to produce 

10 data defining image 290. This technique would work for many images of text, and similar techniques could be 
developed for other text images. 

A variety of text area criteria could be used. For example, a 4 times reduction with a horizontal filter could 
be performed, followed by a morphological horizontal close operation to fill in each line of text and a morpho- 
logical vertical close operation to fill in each column of text. These operations can form a connected component 

15 in a region that includes horizontal lines of text A bounding box can then be obtained around the connected 
component. Graphics can be removed from an Image that includes text 

Data defining image 290 can then be used to extract data defining image 294, an image that includes only 
text 262. This can be done by inverting each pixel in image 290. and then, for each white pixel in the inverted 
image, obtaining the pixel from Image 260 that is in the same position. 

20 The dimensions of the bounding box of image 294 and the dimensions of the largest rectangle that can 

fit within segment 282 can then be used to determine whether image 294 should be scaled to fit within field 
254. If so, an appropriate scaling can be performed in accordance with the techniques described in Scott et 
at. US-A5.097,518. or other conventional techniques. Scaling of image 294 produces data defining image29.6. 
Data defining image 296 and data defining image 250 can then be used to produce data defining image 

25 300. a version of image 250 in which field 302 shows the same sequence of words as text 262 in image 260. 
while the rest of image 300 is the same as the remainder of the form shown in image 250, including field 304, 
which is the same as field 252. Images 296 and 250 can be used, for example, by positioning the upper left 
hand corner of image 296 at the same position as the upper left hand corner of segment 282 in image 280 
and then performing an OR operation between each pixel in image 250 and the pixel in the same position in 

30 image 296, if any. As a result any pixel that is black in image 250 is also black in image 300, but a white pixel 
in image 250 that is at the same position as a black pixel in image 296 is switched to a black pixel, while other 
white pixels remain white. 

Fig. 8 resembles Fig. 7. but shows an implementation in which the criteria and size threshold described 
above are not necessary. 

35 In Fig. 8. image 320 shows a form that includes fields 322 and 324. as in image 250 in Fig. 7. In addition, 

a user has made mark 326 inside field 324 to indicate that field 324 should be filled with text 

Data defining image 320 can be used to obtain data defining image 330. a version of image 330 in which 
pixels within field 324 are all black and pixels outside field 324 are all white. Image 330 could be a mask pro- 
duced by techniques like those described in the image morphology application. For example, a mask for fields 

40 322 and 324 could be produced as described above in relation to image 270 in Fig. 7; then mark 326 could be 
detected based on its extent and curvature, a combined criterion that distinguishes it from small characters as 
well as from straight line features such as the boundaries of fields 322 and 324; a pixel from mark 326 could 
then be used as a seed to fill an inverted version of the mask; the filled inverted mask could be combined 
with the unin verted mask by an AND operation to obtain data defining image 330. Mark 326 could alternatively 

45 be a highlight mark, so that it could be detected using the techniques described in Bloomberg et al.. USrA 
5.048,109. Various other similar techniques could be used. ^.-^ 

Data defining image 320 and data defining image 330 can then be used to obtain data defining image 340, 
a version of image 320 in which all pixels within the area defined by segment 332 in image 330 are white, thus 
removing mark 326. Like image 320, image 340 includes fields 342 and 344. 

50 Image 350 shows text surrounded by mark 352. Mark 352 can be made by a user to indicate that the text 

it surrounds should be used to fill a field in the form shown in image 320. 

Data defining image 350 can be used to obtain data defining image 360, a version of image 350 in which 
pixels within a bounding box around the text surrounded by mark 352 are all black and all other pixels are 
white. Image 360 could be produced by techniques like those described in the image morphology application. 

55 For example, mark 352 could be detected based on its extent and curvature, a combined criterion that distin- 
guishes it from small characters such as those inside it; a mask for the text inside mark 352 could then be 
obtained using a technique like that described in relation to image 290 in Fig. 7, thus obtaining data defining 
image 360. Mark 352 could alternatively be a highlight mark, so that it could be detected using the techniques 
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Reformatting as illustrated in Fig. 10 could also be applied to text that would fit within a field but that is 
formatted inappropriately for the field. For example, text in a narrow column can be reformatted to fit into a 
field that is wide, leaving additional space at the bottom of the field for further text 

5 3. Linking Fields and Text 

In the implementations described above, a single text segment is used to fill a single field. In many appli- 
cations, however, it is desirable to fill a number of different fields with text, where some of the fields may be 
continuations of other fields and where the text may be drawn from different parts of a document. This can be 
10 accomplished by linking each field with text to be used in filling the field. 

Figs. 11-13 illustrate linking in a fill form operation. Fig. 11 illustrates how spacings between paragraphs 
of text can provide information used to link paragraphs to fields. Fig. 12 illustrates how marks in fields and in 
text can indicate continuations between fields and text segments so that a series of fields can be linked to a 
series of text segments. Fig. 13 illustrates how encoded information can indicate features of fields and text 
IS segments so that fields can be linked to appropriate text segments. 

Fig. 11 illustrates a way of indicating text to go in different fields for use with the implementation of Fig. 7. 
Image 530 includes a number of lines of text that are divided into three paragraphs. Paragraphs 552 and 554 
are separated by one space and paragraphs 554 and 556 are separated by two spaces. The spacings between 
paragraphs can be measured as described in EP-A-0,526,196; the technique described there could be applied 
20 to an appropriate segment of the image to measure a specific paragraph spacing. 

A processor that receives data defining an image showing a form with more than one field and image 530 
can use the differences in spacing to determine the Unkings of fields and text The processor can handle each 
field In the order in which It is found on the form and can fill each field with text in the order In which the text 
is found in image 530. The processor can thus link the text in paragraph 552 to a first field. Upon finding one 
25 space between paragraphs 552 and 554, the processor can also link the text in paragraph 554 to the first field. 
But upon finding two spaces between paragraphs 554 and 556. the processor can link paragraph 556 td"a 
second field following the first field. 

Fig. 12 illustrates a way of indicating continuations of fields and of text segments for use with the imple- 
mentation of Fig 8. 

30 Image 570 shows a form that includes machine readable form identifying information 572 that a processor 

can use to obtain data describing the form. For example, the identifying information could be used directly to 
obtain a description of the form or it could be used to obtain data for accessing a stored description of the form. 
The description of the form can then be used to obtain data from the fields of the form. 

Image 570 includes a field for requesting a fill form operation, and check box 574 in this field is marked 

35 to indicate a request Another field includes check box grid 576 for indicating a two digit number of pages of a 
form to be filled, and a larger number could be written in boxes 578 for recognition by a processor if necessary. 

A processor receiving data defining image 570 also receives a first set of images 580 showing a form and 
a second set of images 600 showing a text From data defining image 570, the processor can obtain data in- 
dicating the number of pages indicated by marks in grid 576 and boxes 578. This number can then be used 

40 in finding the separation between images 580 and images 600. 

The processor can obtain data about the fields in images 580 in sequence. Image 582 includes field 584 
with mark 586 indicating that it should be filled. Image 588 includes field 590 with mark 592 indicating that it 
should be filled and also, since It Includes a "c". indicating that it is a continuation of the immediately preceding 
field in the sequence, field 584. Image 588 also includes field 594 and image 596 includes field 598. each 

45 marked to indicate that it should be filled. ■ - 

The processor can then link text from images 600 to the fields. Image 602 includes mark 604 indicating 
that the text segment it surrounds should be used to fill a field, so the text segment surrounded by mark 604 
can be linked to field 584. Image 606 includes mark 608 indicating that the text segment it surrounds should 
be used, and the text segment is also marked with a "c", indicating that it is a continuation of the preceding 

50 marked text segment, so that the text segment surrounded by mark 608 can also be linked to field 584. Image 
606 also includes mark 610 surounding a text segment and image 612 includes mark 614 surrounding a text 
segment, and these text segments can be linked to subsequent fields in the sequence, such as fields 594 and 
598. 

The processor can then use the linking data, together with data defining image sets 580 and 600, to pre- 
ss duce data defining a form with text in the fields. If the text segment surrounded by mark 604 is too large for 
field 584. the overflow can go to field 590, since field 590 is a continuation of field 584. If the text segment 
surrounded by mark 604 does not fill field 584. part of the text segment surrounded by mark 608 can be used 
to fill field 584, because the text segment surrounded by mark 608 is a continuation of that in mark 604. If the 

9 



BNSOOCID: <EP 05e5074A2_L> 



EP 0 585 074 A2 



10 



15 



20 



25 



30 



35 



40 



45 



50 



55 



he .mage morphology apphcahon. Various other similar techniques could be used. RatherJhan usina a 4- to 
^^^F.g^13.llustratesawayof.ndicatingfeaturesoffieldsandoftextsegmentsforu 

X as Shown, so that the processor links It to field 666 In image 662. which rallolf S 

requesT '''' ^^"'^ '--^-^ ^ *ext editor at a user's 

Table I shows linking data that could be obtained for the fields and text segments 



Field ID/Location 



Field 1/Sheet 1 



Field 1 /Sheet 1 



Field 1/Sheet2 



Field 2/Sheet 2 



Field 3/Sheet 3 



Field Boundaries 



x1,y1.w1,h1 



x1.y1.w1,h1 



x2.y2,w2.h2 



X3.y3,w3,h3 



X4.y4.w4,h4 



Text Location 



Page 1 



Page 2 



Page 2 



Page 3 



Text Boundaries 



X5.y5.w5.h5 



x6,y6,w6,h6 



x7.y7.w7.h7 



x8,y8.w8,h8 



Continuation ? 



No 



Yes 



No 



No 



Table 1: Unking Data 
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shown in Fig. 12, including positioning data indicating boundaries of fields. The linking data can be stored In 

any appropriate data structure that permits a processor to access It In automatically obtaining data defining 

an image with text In fields. 

As shown In Table I. the linking data can include a sequence of entries, each including several items of 
5 data. Each entry is illustratively shown beginning with a data Item that Identifies a field and indicates its sheet 

of the form. For field 1. which starts on sheet 1 and continues on sheet 2, the linking data includes entries 

both for the part of field 1 on sheet 1 and also for the part of field 1 on sheet 2. 

Each entry also Includes a data item indicating the field's location, illustratively indicated as the x- and y- 

coordinates of the upper left corner, a height and a width. Each entry then includes data items relating to a 
10 text segment linked to the field, so that field 1 , which Is linked to two text segments, Includes separate entries 

for the segments. 

The data items relating to each text segment include the page of text on which it appears, its location, and 
an indication whether it is a continuation of the text segment in the previous entry. 

A processor using the linking data in Table I to determine how to fill fields first fills field 1 on sheet 1 with 
15 a text segment from page 1. If the text segment does notflll the field, a text segment from page 2 is also included 
in the field. Any overrun is put In field 1 on sheet 2. a continuation of field 1 on sheet 1. Then the remaining 
fields are filled with their respective text segments. 

A data structure like Table I could similarly be prepared for the form fill operation requested In Fig. 13. In 
that case, the types Indicated by the machine readable information are used in obtaining links between fields 
20 and text segments. 

The scaling and reformatting operations described in relation to Fig. 10 can be used in combination with 
fields and text segments that have been linked as Illustrated in Table I. For example. If a field and one or more 
continuations of the field are linked to a series of text segments, the text segments can be used to fill the field 
and its continuations without scaling or reformatting if possible. If the last continuation Is reached and the rb- 
25 malning text segments could not be legible if scaled to fit it, even if reformatted, then it is necessary to scale 
and perhaps reformat previous text segments to reduce the scaling needed in the last field. ^ 

C. Applications 

30 The invention could be applied in many ways to a wide variety of forms and texts. 

The invention is especially useful where a form includes large text fields of varying sizes, into which a user 
desires to fill text. An invention proposal form, for example, may Include a first sheet with many small fields 
and then one relatively large field for description of the invention. The description can be continued In even 
larger fields on subsequent sheets if necessary. In this example, a user could prepare the description of the 

35 invention with a standard text editor, print It, and use the technique of the invention to obtain an invention pro- 
posal form with the description in the appropriate fields. The remaining small fields could be completed by 
hand or with a typewriter. 

D. Miscellaneous 

40 

The Invention has been described in relation to implementations in which images of a form and text, both 
received from a scanner, are automatically merged. The invention might also be implemented with images of 
a form and text that are received from sources other than a scanner. 

The invention has been described in relation to software Implementations, but the invention might be Im- 
45 plemented with specialized hardware. ^ 

The invention has been described in relation to implementations that operate on text In the English lan- 
guage. The invention could readily be extended to text in other languages that follow the conventions of English 
relating to word positioning. The invention might also be extended to text in languages with other conventions, 
such as vertical lines, right-to-left lines, and so forth. 
50 The invention has been described in relation to implementations that fill a field with text, but the invention 

might also be implemented to fill some fields with text and other fields with graphical items, where a graphical 
item could be scaled to fit a field but could not be reformatted. Some types of graphical items could be dis- 
criminated, and techniques could be developed to distinguish other types of graphical items. 

The invention has been described in relation to Implementations in which it Is unnecessary to recognize 
55 characters or words within a text. The invention might, however, be Implemented together with operations that 
perform character or word recognition. 

The invention has been described In relation to implementations that obtain a different version of a text 
that includes words in the same order as In the text, but the Invention might be used In combination with op- 
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erations that modify a text by changing words or changing the order of words in the text. 
Claims 

1. A method of operating a system that Includes: 

image input circuitry for receiving data defining images as input; 
image output circuitry for providing data defining images as output; and 

a processor connected for receiving data defining images from the image input circuitry and for 
providing data defining images to the image output circuitry; 
the method comprising: 

operating the processor to receive input image data from the image input circuitry: the input image 
data defining a first image set that shows a form that includes a first field and a second image set that 
shows a text; ^ 

operating the processor to use the input image data to automatically obtain form image data text 
image data, and positioning data; the form image data defining an image of the form from the first image 
set: the text .mage data defining an image of the text from the second image set; the positioning data 
indicating positions of boundaries around the first field so that the text can be positioned within the first 
Ttelo; and 

operating the processor to use the form image data, the text image data, and the positioning data to au- 
tomatically obtain output image data and to provide the output Image data to the image output circuitry 
the output image data defining a third image set that shows the text and the form, the third image set 
showing the text positioned within the Iroundaries of the first field. 

2. The method of claim 1 in which the first image set includes a plurality of pixels: the act of operatinq the 
processor to use the input image data comprising: 

applying a blank area criterion to pixels in the first image set to obtain blank area data indicating 
a set of pixels that meet the blank area criterion: a set of pixels that meet the blank area criterion being 
likely to be a blank area; and 
30 using the blank area data to obtain the positioning data. 

3. The method of claim 1 in which the first image set shows a mark positioned inside the first field; the first 
image set including a plurality of pixels; the act of operating the processor to use the input image data 
compnsing: 

35 Obtaining mark data indicating a set of pixels in the first image set that each meet a mark criterion- 

a pixel that meets the mark criterion being likely to be in a mark; and 
using the mark data to obtain the positioning data. 

4. The method of claim 1 in which the input image data include first image data defining the first image set- 
^ the first image data including boundary information indicating a boundary of the first field; the act of op^ 

erating the processor to use the input image data comprising an act of using the boundary information 
in the first image data to obtain the positioning data. 

5. The method of any one of claims 1 to 4 in which the second image set includes a plurality of pixels' the 
act of operating the processor to use the input Image data comprising: applying a text area criterion to 
pixels in the second Image set to obtain text area data indicating a set of pixels that meet the text area 
cnterion; a set of pixels that meet the text area criterion being likely to be an area that includes texf and 

using the text area data to obtain the text image data. 

The method of any one of claims 1 to 4 in which the second image set shows a mark positioned to indicate 
he text; the second image set including a plurality of pixels; the act of operating the processor to use 
the input image data comprising: 

obtaining mark data indicating a set of pixels in the first image set that each meet a mark criterion; 
a pixel that meets the mark criterion being likely to be in a mark: and 
using the mark data to obtain the text image data. 

The method of any one of claims 1 to 4 in which the input image data include second image data defining 
the second image set; the second image data including text information indicating the text; the act of op- 
erating the processor to use the input image data comprising an act of using the text information in the 
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second image data to obtain the text image data. 

The method of claim 1 in which the first image set includes a plurality of fields, the plurality of fields in- 
cluding the first field; the input image data further including field order information indicating an order of 
the fields that begins with the first field and continues to a second one of the fields; the text exceeding 
the first field in size; the act of using the input image data comprising an act of using the field order in- 
formation In the input image data in obtaining first field image data and second field image data; the first 
field image data defining an image of a first part of the text that can fit within the first field; the second 
field image data defining an image of a second part of the text that can fit within the second field; the 
text image data including the first field image data and the second field image data. 

The method of claim 1 in which the first image set includes a plurality of fields, the plurality of fields in- 
cluding the first field; the text including a plurality of segments; the input image data Including linking in- 
formation Indicating, for each of the fields, a respective one of the segments; the act of operating the proc- 
essor to use the input image data comprising: 

using the linking information to obtain first field data indicating that a first one of the segments is 
a respective segment of the first field; and 

using the first f ield data and the positioning data to obtain the text image data so that the text image 
data define the first segment of the text so that it can fit within the first field. 

The method of claim 1 in which the second image set shows the text with a first shape; the positioning 
data indicating boundaries of the first field having a second shape different than the first shape; the text 
including a word that has a first position within the first shape of the text; the act of operating the processor 
to use the input image data comprising an act of using the input image data and the positioning data to 
obtain the text image data so that the text image data define an image showing the text with the second 
shape and so that the word has a second position within the second shape of the text, the second position 
being different than the first position, ,,7 

11. The method of claim 1 in which the second image set shows the text with a first scale; the positioning 
data indicating boundaries of the first field into which the text with the first scale cannot fit; the act of 
operating the processor to use the Input Image data comprising an act of using the input image data and 
the positioning data to obtain the text image data so that the text image data define an image showing 
the text with a second scale different than the first scale so that the text with the second scale can fit 
into the boundaries indicated by the positioning data. 

12. The method of claim 1 in which the input image data define a sequence of images that includes the first 
image set and the second image set; the first image set preceding the second image set in the sequence 
of images; the input image data including separation information indicating a separation between the first 
image set and the second image set; the act of operating the processor to use the input image data com- 
prising an act of using the separation information in the input image data to separate the first Image set 
and the second image set 

13. The method of claim 1 in which the form has a first skew direction in the first image set and the text has 
a second skew direction in the second image set; the act of operating the processor to use the input image 
data comprising an act of using the input image data to obtain skew correction data indicating a skew cor- 
rection that Is necessary to make the first skew direction and the second skew direction the same; the 
act of operating the processor to use the form image data, the text image data, and the positioning data 
comprising an act of using the skew correction data so that the third image set shows the text and the 
form at the same skew direction. 

14. A system com pris in g : 

image Input circuitry for obtaining data defining images as Input; 

image output circuitry for providing data defining images as output; 
memory storing data; and 

a processor connected for receiving data defining images from the image Input circuitry and for 
providing data defining images to the image output circuitry; the processor further being connected for 
accessing the data stored in memory; 

the data stored in the memory indicating: 

input instructions the processor can execute to receive input image data from the image input cir- 
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cuitry: the input image data defining a first image set that shows a form that includes a field and a second 
image set that shows a text; lu 

data obtaining instructions the processor can execute to use the input image data to automatically 
obtam form image data, text image data, and positioning data; the form image data defining an Image of 
the form from the f ,rst .mage set; the text image data defining an Image of the text from the second image 

H TJ- !i "^""^ °^ boundaries around the field so that the text can be posi- 

tioned withm the field; and *^ 

. H V processor can execute to use the form image data, the text image data 

and t he positioning data to automatically obtain obtain output image data and to provide the output image 

^n?/hl f ""ff ""^9^ ''^'^ '"'"9 « '^"^ '-"^Se set that shows the text 

and the form, the third image set showing the text positioned within the boundaries of the field. 

1 5. An article of manufacture for use In a system that includes: 

image input circuitry for obtaining data defining Images as Input; 
« ''"39® oulpui circuitry for providing data defining images as output; 

memory for storing data; 

a storage medium access device for accessing a medium that stores data- and 
.rr.. Jr.r^^Tl T^ce>i^mg data defining images from the Imag^ input circuitry and for 

providing data defining .mages to the image output circuitry; the processor further being connected for 
20 receiving data from the storage medium access device; 

the article comprising: 

i, ,..J'^T ^^"^T '"^'^"'^ ^oce^^6 by the storage medium access device when the article 

IS used in the system; and 

tho , '^^^TT^m '"^ ''^'^ "'^'^'"'^ ""^^ a=«=ess device can provide 

the stored data to the processor when the article is used in the system; the stored data indicating- 

'"P"' 'nstruotions the processor can execute to receive input image data from the image input clr- 

^^r'' .T^T ''^''"'"^ ^ '"^^9^ ^ '"'"^ tl^^t '"'^'"d*^ a field and a second 

(mage set that shows a text; 

oh»«io f^',!' °'"^'"'"9;"«[^"=«0"s the processor can execute to use the input image data to automatically 
obtain form .mage data, text .mage data, and positioning data; the form image data defining an image of 
the form from the f.rst .mage set; the text image data defining an image of the text from the second image 

HoLh T'? m """^ °^ boundaries around the field so that the text can be posi- 

tioned withtn the field; and y^^*- 

.nH ^ ^he processor can execute to use the form image data, the text image data 

and the positioning data to automatically obtain obtain output image data and to provide the output image 
date to the tnage output circuitry; the output image data defining a third image set that shows the text 
and the form, the third image set showing the text positioned within the boundaries of the field 
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(^ Automatic image creation by merging text image and form image. 

(57) Input images of a fonm (250) and of a text 

(260) are used to automatically obtain an output ; 
image (300) showing the form with the text in a 
field (302). A processor receives data defining 
the input images (250.260), automatically ot>- 
tains positioning data indicating positions of 
boundaries around the field (254) so that the 
text (262) can be positioned within the field, and 
automatically obtains data defining the output 
image. The positioning data can be automati- 
cally obtained by operating on plain images of 
the fonr^ and of the text ; by operating on 
images of the form and of the text that have 
been marked to indicate the field and the text; 
or by operating on images of the fonm and of the 
text that include machine readable infomnation 
indicating the positions of the field and the text 
Skew can be corrected before merging a text 
with a form, and text can be scaled (294,296) 
and reformatted if necessary to fit into a field. If 
the input images include more than one field on 
the fomn and more than one text segment, the 
processor can obtain linking data indicating a 
respective text segment for a field. The linking 
data can also indicate a field that is a continua- 
tion of another field or a text segment that is a 
continuation of another text segment, in case of 
overrun or insufficient text to fill a field. The 
processor could be in a fax server, a digital 
copier, or a computer with a scanner and a 
printer. 
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